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Abstract 

Multi-label active learning is a hot topic in reducing the label cost by optimally choosing the 
most valuable instance to query its label from an oracle. In this paper, we consider the pool- 
based multi-label active learning under the crowdsourcing setting, where during the active query 
process, instead of resorting to a high cost oracle for the ground-truth, multiple low cost imperfect 
annotators with various expertise are available for labeling. To deal with this problem, we 
propose the MAC (Multi-label Active learning from Crowds) approach which incorporate the 
local influence of label correlations to build a probabilistic model over the multi-label classifier 
and annotators. Based on this model, we can estimate the labels for instances as well as the 
expertise of each annotator. Then we propose the instance selection and annotator selection 
criteria that consider the uncertainty/diversity of instances and the reliability of annotators, such 
that the most reliable annotator will be queried for the most valuable instances. Experimental 
results demonstrate the effectiveness of the proposed approach. 

Key words: Multi-Label learning, active learning, weak supervision, crowdsourcing 


1. Introduction 


Multi-label learning has received sig nifican t attention to deal with examples that are associated 
with multiple classes simultaneously ZZ14l | . It is worth noting that it is usually quite expensive 


to obtain the labels for training, as every possible label has to be checked whether it is proper 
for the instance. Thus multi-label active learning, which reduces the labeling c ost by actively 


selecting the most valuable instances to query becomes a hot topic 


LGl 
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HZ1J |. 
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Most multi-label active learning algorithms rely on the domain expert, or an oracle, to provide the 
ground-truth label for each query. While the labeling cost of domain experts is high, by employing 


multiple low cost non-expert labelers, crowdsoucing provides a low cost way to get annotations . 


A number of studie s exploiting the wisdom of crowds have arisen recently 


WBBPinl . 


SOJNQ8, 


RYZ+ld . 


ZLPM14| , mainly focusing on single-label tasks where each instance is associated with 


only one label. Exploiting the wisdom of crowds for multi-label data has barely been touched to 
the best of our knowledge. 


In this paper, we consider the problem of exploiting the wisdom of crowds for multi-label tasks. 
While multiple annotators may be available, in real-world applications, the labeling cost would 
still be high for multi-label objects considering that every label needs to be checked on each 
instance by several annotators, whereas the annotation budget is often limited. Thus we define our 
problem from the active learning perspective, where instances are actively selected to query the 
supervised information, but rather than a high cost oracle providing the ground-truth, multiple 
imperfect annotators are available for labeling. The goal of our work is to learn an effective 
multi-label classifier with as less annotation cost as possible. 


By decomposi ng the mul ti-label task into a series of independent binary classification problems. 


Y. Yan’s work 


YRFDll[ | can be applied to each label independently, neglecting the fact that the 


information of one label may be helpful for learning another related label, especially when there 
are insufficient training data for every label. To handle the problem of multi-label active learning 
from crowds, we propose the MAC (Multi-label Active learning from Crowds) approach, which 
incorporates the local influence of label correlations to build a probabilistic model for multi-label 
classifier and annotators. Based on this model, we can estimate not only the labels of instances 
but also the expertise of annotators. Then we propose our instance selection criterion which 
considers both uncertainty and diversity of instances, and the most reliable annotator on the 
most valuable instance is queried. 


In the following we start with a brief review of some related work. Then, we propose our MAC 
approach and report the experimental results. Finally, we conclude the paper . 
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2. Related Work 


Multi-label Active Learning Multi-label learning deals with examples that are associate d with 
multiple labels simultaneously; it has received significant attention during the past decades 


and achieved successf ul appli cation in various tasks s uch as i mage annotation 


function classification 


EWOll ] and text categorization McC99|. To collect labels for multi-label 


BIjSBOJ], gene 


ZZ14 | 


training, each of the multiple labels should be checked whether it is proper for an instance; the 
labeling cost is quite expensive. Thus multi-label active learning, which reduces the labeling 
cost by actively selecting the most valuable instance to query from an oracle, has attracted great 
attention. Existing multi-label active learning research mainly focus on designing the criterion 
for in stance s election, and can be roughly categorized into three categories; 1) uncertainty sam¬ 
pling SCC08l | which selects the instance that the classifier is most uncertain about ; 2) expected 


loss reduction 


HLllI ] which querie s instance th at minimizes the expected loss of the classifier; 


and 3) com bining multip le criteria 


most work 


HTdl 




HZ13l | to select instance. Given the selected instance. 


LG13I ] query all the labels for the instance, which may lead to information 


redundancy and wasting of oracle’s effort, since labels are often correlated in multi-label learning. 

Crowdsourcing With the advent of crowdsourcing platforms such as Amazon Mechanical Turk 
(AMT), crowdsourcing which makes use of crowdsourced annotatio ns from m ultiple imperfect 


lab elers is w idely used for t asks includ ing sentiment classification 
sis 


RYZilOl, image tagging WBBPiril | and webpage categorization ZLPM14l |. As annotators 


SOJN08 I. medical diagno- 


may have different expertise on different tasks, the common wisdom is to distribute tasks to 
multiple annotators and then estimate the correct labels via some aggregation schemes. Simply 
taking the majority voting without considering the annotators ability variance may lead to poor 
impact on subsequent learning. Hence, in order to solve the above problem of annotators, a 
number of methods assessing abilities of annotators have been proposed. The annotator’s ex¬ 


pertise o n specific task are 


accuracy 


WRW+nd 


dimensional vectors 


usually modeled by eit 


ler measures with explicit explanation like 


KOSllll a nd confusion matrix 


BYZ+inl . 


ZLPM14| . or complex high multi- 


WBBPlOt]. A few work trying to get 


crowds cost are proposed, for example, 


YRFDll 


good learning performance with low 


FYT14i | consider the cost by adapt ively as- 


signing the best annotator to specific task under the active learning framework and 


ERH14l | 


propose an approach to approximate the performance of majority voting with less annotations. 
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While current work on crowdsourcing mainly focus on single-label tasks, to our best knowledge, 
using crowds in a cost economic way has not been studied in multi-label learning. We are 
presenting possibly the first approach to exploiting the wisdom of crowds efficiently in multi¬ 
label learning. 


3. MAC: Multi-Label Active learning from Crowds 

We consider the pool-based multi-label active learning from crowds, where a set of labeled an¬ 
notated data of Ni examples Di = {(xi, zi, {yij}),... , (xvj, zw; j {yAfij})} and an unlabeled set 
of Nu examples Du = {xwi+i, ■ ■ ■, xVi+ATu} are available for training. Each instance Xj is a 
d-dimensional feature vector. Given an instance Xj G Di, Zj G is its true label as¬ 
signment, where L is the number of labels and z[ = 1) indicates that the /-th label is tagged 

as positive(negative) for x*; and yij G —1,0}^^^ is the label assignment given by annotator 
j, where ylj = 0 means that the annotator j gives no annotation for Xj on the Z-th label , and 
y\j = -|-1(—1) means annotator j tags the Z-th label as positive(negative) for Xj. We assume a set 
of M annotators {wj}j where j G {1,2,... ,M} are available. We do not require each instance 
to be annotated by all annotators on every label, and the annotator set labeling instance Xj on 
label Z is denoted as M{, i.e., = {j\y[j = 1(—1)}. During the active learning process, rather 

than an oracle being available to provide the groundtruth labels z, multiple imperfect annotators 
with various expertise are available for annotating. The target of our work is to actively select 
the most reliable annotator for the most valuable instance such that we can learn an effective 
multi-label classifier with as less labeling cost as possible. 

Compared with traditional multi-label active learning, we can see that the key challenges in the 
active learning from crowds lie in that: 1) the annotators reliability can be various for specific 
instance, which requires the careful selection of the annotator with the best expertise for a query; 
2) the labels provided by selected annotator for specific queried instance can be noisy, which 
requires a further aggregation step to get a better estimation of the groundtruth label. To 
solve the above challenges, we use a probabilistic model which can simultaneously inference the 
annotators expertise and groundtruth label at the same time. Furthermore, the local influence of 
neighborhoods’ label correlations are incorporated in this probabilistic model to help the multi¬ 
label classifier learning. After we obtain the estimation of annotators expertise and instances 
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labels, we can design some criterion for instance and annotator selection to get the most reliable 
annotation for the most valuable instance. In the following subsections, we first propose our 
probabilistic multi-label crowdsourcing model and then the active selection strategy. 


3.1. Probabilistic Multi-Label Crowdsourcing Model 


We first introduce our probabilistic model on one single label to describe the classifier and anno¬ 
tators, and then encode the local influence of neighborhoods’ label correlations in this model to 
deal with multi-label crowdsourcing. 




Figure 1: (a) The single label graphical model for instances {xi}, annotations {ylj} and unknown groundruth {z*}; 
(b) Our multi-label graphical model for instances {xi}, {xi}, annotations {ylj} and unknown groundruth {z*} on 
label 1. 


Figure 1(a) illustrate the probabilistic graphical model on label I over the instances {xj}, the 
observed annotations given by annotators {y^j}, and the unobserved groundruth labels {z(}. We 
assume that the annotations given by the annotators depend both on what input they observe and 
what task they are expected to finish, i.e., the input instance x and the target groundtruh label 
z. The groundtruth label of the instance is believed to reflect specific property of the instances, 
for example, Zj = 1/ — 1 may denote the presence/absence of the tree in a picture. Using 9 
to denote the parameters involved in this model and Pr{9) its prior probability distribution, the 
probabilistic graphical model can be represented using the following joint probability distribution: 


i j&M‘ 


( 1 ) 


For each annotator j, we define an expertise level variable eh to build the relationship between 
its annotation yh and the input (xj,z(), which is mathematically define as the probability that 
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annotator j provides the groundtruth as its annotation for instance i. Here we exploit a logistic 
sigmoid acting on some function over the instance Xj, that is, 

4j = P{y\j = z'lxi, 0) = ( 2 ) 


where the logistic sigmoid function is dehned as a(z) = 1/(1 + exp{—z)). Using /(•) to denote 
the indicator function, then the distribution p(y(j|z(,Xj, 0 ) can be formulated as the following 
Bernoulli distribution. 




ijj 


(3) 


Since we are dealing with classification, we use a logistic sigmoid acting on some function /g q 
over instance Xj to model the Bernoulli distribution p(z(|xj, 9) for z, that is, 

p(z-= l|xi, 6 ») = cr(/o 0 (xi)) and p(z-=-l|xi, 6 ») = cr(-/o^(xi)). (4) 

While any function can be used to implement /g g and /j g, for ease of exposition, we consider the 
linear discriminating functions, i.e., /^^(xj) = (wg) Xj and /jg(xj) = (wj) x*. Given this, the pa¬ 
rameters becomes the classifier/annotator parameters 6 = {wQ,{wj}}. To overcome overhtting, 
we introduce a zero-mean A-variance Gaussian prior for {wg} and {wj}, i.e., : 


p^( 6 ») =p(wo|A) p(’wj|A) where p{wq\X) = p{wj\X) = J\f{0, X ^1) (5) 

j&Ml 

We can estimate the parameters by exploiting the ma ximum l ikelihood criterion, to solve which a 
standard Expectation-Maximization (EM) approach DLRTtI can be used with missing variables 
{z(} and observed variables {y[j}ij- 


E-step: Given current estimation of the parameters 9 = {wg,{wj}} from last M step, the 
posterior probability of ground truth label {z(} is computed: 

p(z() = p(z^|xi, {w^}) 

ocp(z^|xi,w[)) p(y(^.|z^,Xi,w'). (6) 

j&M\ 

Substituting Eq. [3] and Eq. [Hinto Eq. El E-step reduces to: 

p(z( = 1) oc o-(w[) Xi) CT(wj Xi) JJ o-(-wj Xi) (7) 

y\ .=1 .=—1 

j -ij j 

p{z[ = -1) oc ct(-w[, Xi) Yl n X*) (8) 

=—1 y^. =1 

^ j -ij 
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M-step: To estimate the parameters 0 = {wq, {w* }}, we maximize the expectation of the joint 
log-likelihood of over parameters 6 = {wg,{w^}}, with respect to the posterior 

probabilities of {z\} computed by last E step: 

6 = argmaxQ(0), 

0 

where Q{9) is 

Q(6») = E^[lnP{{y\j}ij,{z[}i\{yL,}i,\v\^,{\v^j})pr{e)] 

= Ez[ln]Jp(z^|xi,w[))p(w|)|A) Ylp{yij\z\,Xi,{w^j})p{w^j\X)] 
i j 

i ij 



j 

Substituting Eq. [3]and Eq. [4]into Eq. [U Q{6) reduces to: 

Qi^o) = Xi)p(zi = 1) - + exp(w|) Xi)]. (10) 

i i 

Qi^j) = = 1 ) • yij - Y1 

i i 

The gradient of Q with respect to parameters Wq,w*- is computed as: 

^ = 1) - f^(wo Xi)]. (12) 

i 

^ = 1) - X* • y^^.)] • y\j- (13) 

i 

To deal with multi-label data, one straightforward approach is to treat each label independently 
and applying the above model on each label separately. But by this, the label correlations are 
ignored, which however is widely verified to be helpful for multi-label learning. In this paper, 
we derive a local code to enhance the feature representation of instances, which reflects the local 
influence of its neighborhoods’ label correlations. We use the simple idea that instances similar 
in the feature space should also be similar in the label space, and utilize the information from 
the label space of labeled instances to construct extra features. In detail, for each instance x, its 
code vector c is constructed as the label average mean of its k nearest neighbors in the initial 
labeled training set, i.e., 

c = i ^ z,. (14) 

xieAf(x) 
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After the code vector is computed, the enhanced representation of instance x becomes x = [x, c]. 
Then we learn the above probabilistic model on each label separately, but using different instance 
representations for classifier and annotators. For label I, the multi-label classiher parameter wq 
is learned on the enhanced representation, but for the annotator model parameters {wj}, the 
original representations x are used, i.e., wq is of dimension {d + L) and {wj} are of dimension 
d. Utilizing the local influence of label correlations, our multi-label probabilistic crowsourcing 
model can be described by Figure [1(b) [ The reason we use different instance representations for 
the classiher and annotators is that, the enhanced representation of instances are deduced from 
the initial labeled data, which are generated by the true multi-label data distribution, so it is 
expected that only for cases where the training data are consistent with the multi-label data 
distribution, the enhanced representation are to be helpful for learning. While the groundtruth 
estimation aggregated from the multiple annotators should be trustable and consistent with the 
multi-label data distribution, the annotations provided by each annotator can be of low quality 
and far away from the multi-label data distribution, thus the enhanced representation generated 
from the labeled training data would be inconsistent with its labeling behavior. Though here the 
way we compute the enhanced representation is simple, we believe the idea of constructing new 
representations for the classifier using information from the label space would be a good direction 
for multi-label crowdsourcing. 

3.2. Active Selection 

Based on the above model, we can learn an estimation of the multi-label classifier, the annotators 
expertise and the labels for instances from the crowds annotations. In traditional multi-label 
active learning, most work select the instance and query its supervised information on all labels, 
which may lead to information redundancy and wasting of annotation effort since labels are often 
correlated in multi-label learning. To avoid information redundancy, we exploit the idea of first 
choosing the most valuable instance according to its label estimation, and then selecting its most 
uncertain label to query the supervised information. Considering that the supervised information 
in our problem are provided by crowds whose reliability on different instances may be different, 
therefore, the selection of the most reliable annotator for the specific instance-label pair should 
also be carefully considered. 
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Instance-Label Pair Selection We design our instance selection criterion by combining the 
uncertainty criterion with the diversity of queried annotations. LCI (instance Label Cardinality 
Inconsistency) is a commonly used instance uncertain measure in multi-label active learning and 
its combinatio n with other m easures to conduct instance selection has shown promising results in 
several works 


LGl.l 


HZ13l |. It is defined as the inconsistency between the number of predicted 


positive labels of instances and the average label cardinality on the labeled data, 

L . Ni L 

Lci{^) = (Y.m = 1) -jtXEM = i))t 

1=1 '■ j=i i=i 

where z( is the label estimation of instance Xj on 1. We extend LCI to incorporate the query 
diversity during the active process and define a new criterion 

L Ni L 

1=1 '■ j=ii=i 


i* = argmaxC'/(xj) = 


(15) 


max{^,anno{xi)} ’ 

where we set z( = 1(—1) if its probability predicted by Eq. [6]is larger(no more) than 0.5, anno(xj) 
is the number of queried annotations of instance Xj and ^ G (0,1) is a small constant to avoid 
zero devisor. We use the number of queried annotation to tradeoff LCI to avoid querying the 
same instance, and giving less queried instance more chance to be queried which may contain 
more unknown information. ^ G (0,1) is set as 0.5 in this paper. 


Given the selected instance x* by Eq. [T^ the label whose estimation is most uncertain is selected: 

I* = argminC'L(xj*, Z) = | p{z[, = l|xj*) —0.5 |. (16) 

Annotator Selection Given the selected instance and label (x^.,/*) by Eq. [15] and Eq. [161 we 
need to select the most reliable annotator on it to collect high quality annotations. Eq. [2] provides 
the information about how reliable each annotator would be on each instance and label, which 
is inferred from observed annotations by our crowsourcing model. So we compute the expertise 
level of all annotators on the selected instance-label pair and select the most reliable one: 

f = argmaxC'A(xi*,/*,wj‘) = = o-((w'*)'xi*). (17) 

After the instance-label pair and the best annotator on it are selected, the annotation is queried 
and added to the training data to update the multi-label crowdsourcing model. The MAC (Multi¬ 
label Active learning from Crowds) algorithm is summarized in Algorithm 1. 
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Algorithm 1 The MAC Algorithm 
1: Input: 

2: training set D = {Di,Du}, parameters A, k 

3: Train: 

4: initialization 

5: get the enhanced representation x = [x, c] for instance x G L); by Eq. [TT] 

6: get the initial estimation of {wq,w^} using LIBLINEAR on Di 

7: repeat: 

8: get the label estimations for instances x in by Eq. [6] with {wq} and {wj} 

9: select instance Xj* by Eq. [15] 

10: select label I* for instance Xj* by Eq. [T6| 

11: select annotator j* for instance Xj* on label I* by Eg. [T7I 

12: get the annotation for Xj* on label I* from annotator j* 

13: remove annotator j* for instance Xj* on label I* 

14: remove label I* from instance x** if all annotators on I* are removed 

15: remove instance Xj* from Du if all labels of Xj* are removed 

16: add Xj* to Di if Xj* is not in Di, add yj*j*^ to Di for Xj* on label I* 

17: update {wQ,wj} on Di 

18: util the maximum number of queries is reached or Du is empty. 

19: Test: 

20: for instance Xj, get its enhanced representation Xj = [xj,Cj] by Eg. fTTI 

21: get its prediction on label I by Eq. 0] 


3.3. Computational Complexity 


The computa tional com plexity of MAC composes of three parts: 1) the initialization step, we use 


LIBLINEAR 


FCH~*~08l | to get the initial estimation of classifier and annotator parameters which 


is fast and accurate; 2) the active selection part by Eq. 15-17, which is linear in the number of 
instance, the number of labels and the number of annotators; 3) model updating, after the queried 
annotation is added to the training data, the crowdsourcing model is updated using EM. The 
parameters are initialized by results from last step which makes the EM converges fast, usually in 
less than 100 iterations. The computational complexity of E-step in Eq. [6|is linear in the number 
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of involved instances and the number of involved annotations; the computation complexity of 
M-step in Eq. [9] depends on the employed optimization approach. We exploit gradient ascent to 
solve the M-step, which needs to iteratively compute the gradient of parameters until convergence 
or maximum iteration number is reached. At each iteration, the computation complexity is linear 
in the number of involved annotators, the number of involved instance and the number of involved 
annotation. 


4. Experiments 


In this section, we compare our MAC approach with a number baseline methods on two natural 
scene classification datasets in which each picture can be categorized into one or more classes. 


The Image dataset 


BLSBOJll contains 2000 natural scene images and 5 possible labels {desert, 


mounta ins, se a, sunset, trees}, each image having on average 1.24 ± 0.44 labels. The Scene 


dataset 


ZZ07 


contains 2407 images and 6 possible labels {beaeh, field, foliage, mountain, sunset. 


urban}, each image having on average 1.07±0.26 labels. The images belonging to more than one 
class in these two datasets comprise about 22% of the whole data. 


The parameter A is set as le“^ and k is set as 10 in the experiments. As there exists no 
other method for the problem of multi-label active learning from crowds, we compare our MAC 
approach with the following 2 groups of 7 baselines. 


Groupl exploits the label correlations of multi-label data, i.e., the multi-label classifier and anno¬ 
tators are learned respectively on enhanced and original instance representations: 1) MCR±RD: 
learn the classifier and annotators using our multi-label crowdsourcing model in Figure |l(b)[ 
randomly select (instance, label, annotator) for labeling; 2) MV±ACT: instead of learning by 
our multi-label crowdsourcing model, use majority voting on the collected annotations and train 
a logistic regression classier; select the (instance, label) by our active selection component in 
section 3.2, randomly select one annotator for labeling; 3) MV±RD: instead of learning by our 
multi-label crowdsourcing model, use majority voting on the available annotations and train a 
logistic regression classier; randomly select (instance, label, annotator) for labeling; 


^ http://mulan.sourceforge.net/datasets.html 
^ http://cse.seu.edu. cn/people/zhangml/Resources.htm 
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Group2 does not exploit the label correlations of multi-label data, i.e., both the multi-label 
classifier and annotators are learned on original instance representations: 4) SCR-I-ACT: learn the 


classifier and annotators using the probabilistic model in Figure 1(a) on each label independently; 


actively select (instance, label, annotator) by our active selection component in section 3.2; 5) 


SCR-I-RD: learn the classifier and annotators using the probabilistic model in Figure 1(a) on each 


label independently; randomly select (instance, label, annotator) for labeling; 6) SMV-I-ACT: use 
majority voting on the available annotations and train a logistic regression classier; select the 
(instance, label) by our active selection component in section 3.2, randomly select one annotator 
for labeling; 7) SMV-I-RD: use majority voting on the available annotations and train a logistic 
regression classier; randomly select (instance, label, annotator) for labeling. 

3 labelers with different expertise are simulated to generate the crowdsourcing annotations. For 


each dataset, on one specific label I, we proceed as fol 
for this label on the whole dataset by LIBLINEAR 


ows: we f irst train a logistic regression model 


FCH'*~n8l ], then according to the probability 


output of the logistic regression model on the dataset, we cluster the data into three subsets by 
k-means. After that, each of the 3 simulated annotators i, i = 1,2,3 performs as an expert on the 
Ath cluster and gives the groundtruth label for annotation; for the remaining data belonging to 
the other clusters, its probability of correctly annotating the groundtruth label is 75% (the labels 
for these data are randomly switched with probability 25%). By such an annotation generation 
process, the annotations provided by crowds for specific label is dependent on the semantic of this 
label. Furthermore, on each label, the expertise of annotators are different on different instances. 

For each dataset, we randomly partition the data into three parts which compose 2.5%, 47.5% 
and 50% of the whole dataset to respectively construct the initial annotated labeled training data, 
the unlabeled training data and the test data. The random partition is repeated for five times 
on each dataset and the average performance is reported. At each active query, the (instance, 
label, annotator) triple is selected to get the annotation and then added into the annotated data. 
After every 200 annotation queries the performance of the multi-label classifier on the test data is 
reported. The query process terminates after the number of q ueried annotations reaches 8, 000. 
The commonly used multi-label performance measure micro-Fl ZZ14l | is used in our experiments. 


In Figure 2(a)/3(a) and Figure 2(b)/3(b), we test our crowdsourcing learning component and 


the active selection component between methods within Goupl and Group2. In Figure [2(c)[^3(c)| 
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we test the label relationship role by comparing our MAC with the best baseline in Groupl and 
Group2. 


From Figure 2(a) and Figure [2(b)[ we can see that: 1) compared to random sampling, our active 
selection component clearly helps improve learning for both our crowdsourcing learning model 
and majority voting (MAC vs MCR+RD; MV+ACT vs MV+RD; SCR+ACT vs SCR+RD; 
SMV+ACT vs SMV+RD); 2) compared to majority voting, our crowdsourcing model aggregates 
crowds better, especially when the annotations quality can not be guaranteed (MCR+RD vs 


MV+RD; SCR+RD vs SMV+RD). From Figure 2(c) in which MAC is compared with the best 
method in Groupl (MV+ACT) and Group2 (SCR+ACT, SMV+ACT), it can be seen that 
utilizing the label relationship for classifier learning greatly boost learning (MAC vs SCR+ACR; 
MV+ACT vs SMV+ACT), especially at early stage when the number of training data is small. 
Similar results for our MAC and baselines on Scene dataset are also obtained in Figure [3l R’s 
not surprising that by utilizing the label relationship and active selection, our MAC approach 
achieves the best learning performance. 


Image 


Image 


Image 



number of queried annotations 


2000 4000 6000 8000 

number of queried annotations 



number of queried annotations 



(a) Groupl Comparison 


(b) Group2 Comparison 


(c) Gronpl v.s. Group2 


Figure 2: The average results over 5 runs in terms of micro-Fl(the larger the better) on Image. 


Scene 



number of queried annotations 


Scene 



number of queried annotations 


Scene 



number of queried annotations 


(a) Groupl Comparison 


(b) Group2 Comparison 


(c) Groupl v.s. Group2 


Figure 3: The average results over 5 runs in terms of micro-Fl(the larger the better) on Scene. 
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5. Conclusion 


In this paper, we consider the problem of reducing the labeling cost of multi-label learning 
by actively learning from crowdsourcing annotations, where during the multi-label active query 
process, rather than resorting to a high cost oracle for the ground-truth, the labels are provided 
by multiple low-cost imperfect annotators. To deal with this problem, we propose the MAC 
(Multi-label Active learning from Crowds) approach which encodes the local influence of label 
correlations to build a probabilistic multi-label crowdsourcing model and then propose an active 
selection criterion to query the best annotator for the most valuable instance. Experimental 
results show that our approach is more effective than baselines. Currently, the computational 
complexity of MAC is linearly dependent on the number of labels; in the future, we plan to 
develop more efficient methods for large number of labels. 
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